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(54) Method and system for maintaining freeform ink annotations on changing views 



(57) The method and system of the invention main- 
tains the indication of the logical relationship of freeform 
digital ink annotations to objects in an electronic display 
even as the view changes. The invention maps the log- 
ical relationship of the freeform digital ink annotations 
to objects in the view based upon the layout relation- 



ships of the freeform digital ink annotations to the ob- 
jects in the view. If the view changes, the invention main- 
tains the layout relationship of the freeform digital ink 
annotations to the objects in the view based upon the 
mapping of the freeform digital ink annotations to the 
objects. 
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[001 3] Dynomite does not permit annotations directly on the view and the annotations are not rendered on the new 
views in accordance with changes to the views. 

[0014] The method and system ot the invention maintain the layout relationships between freeform digital ink anno- 
tations and objects h changing views. As a user makes a treeform digital ink annotation on a view and as the view 
changes the method and system of the invention adapt the marks in accordance with the changes. Thus, the logical 
relationship between the treeform digital ink annotations and the objects made in one view can be understood in future 
views based upon the layout relationships. 

[00151 The method and system also maintain the indication of the logical relationships between freeform digital ink 
i annotations and objects in dynamic media such as video and interactive virtual worlds. For example if a user who is 
exploring an information space with a hyperbolic browser makes a freeform digital ink annotation on a node of the 
tree when the user changes thedisplay to bring another portion of thetree into focus, the freeform digital ink annotation 

* 1°^ ^ ?S 9 With the,fee tocontinue to indteate th * log**' relationship of the freeform digital inkannotation 
to tne node. 

5*??. J" 8 methods andsystems of this invention maintain the indication of the logical relationships between freeform 
SSLHif? r S 0bjeC,S 8 Vi8W as tne view * *• annotat «l objects changes. This invention brings the 

S SEStZSL t0 dynamiC media ' chan9in9 w °«H>'ocessor documents, information visualizations, 

virtual worlds, digital video and word processors. 

[0017] The system and method of the present invention maintain the indication ot the logical relationship even if the 
objects that are logically related to the freeform digital ink annotations are delsted. inserted, copied or moved or oth- 
" ^T 5 " k T m0d and SyStem ateo maintein 100 indication of the logical relationship it the layout presentation 
o the objects in the display change because the objects reftow. scale, resize, rotate, stretch, distort or morph or if any 
r*ii« ^f^t th * layout presentation changes, regardless of whether these changes occur discretely or continuously 
SS? T^H?* ano ! her exemplary embodiment of the systems and methods of the invention provides navioal 
™^ r , E? T . di9tel ink annotation a P<> weriul ,00 ' f °' navigating dynamic documents. A useffe 

^n^l ^ ! ! r8e ! 0rm di9ital ink annota,ion in an art >»«-y view, and the display is returned to the state it was 
in when the selected freeform digital ink annotation was made. 

whraL Pre,erred embodime nts o» this invention will be descrbed in detail, with reference to the following figures, 

Fig. 1 is a block diagram of an embodiment of the system for displaying changing views of this invention 
Fig. 2A shows a display of an annotated hyperbolic browser in accordance with an embodiment of the'present 

invention; 

Stf B ^° W !!i! ,e annotated n ype*o»ic browser of Fig. 2A with the view changed and freeform digital ink annc- 
tatoons adjusted in accordance with the change in the view in accordance with an embodiment of the invention- 
Fig. 3A shows a display of an annotated document; • 

Fi& 3B shows the annotated document of Fig. 3A after modification of the document and shows the adjustments, 
to the freeform digital ink annotations in accordance with an embodiment of the present invention- 
Figs. 4A-4F show a flowchart outlining a control routine for mapping freeform digital ink annotations to objects in 
a display m accordance with an embodiment of the prevent invention; 

Figs. 5A-5B show a flowchart outlining a control routine for rendering freeform digital ink annotations on a changed 
view in accordance with an embodiment of the present invention; and 

Fig 6 shows a few examples of different types of freeform digital ink annotations. These and other features and 
advantages of this invention are described in orare apparent Irom the following detailed description of the preferred 
emooaiments. 

!?^L! 19 ' 1 Sh ° WS ° n ® eXemplary embodiment of the electronic system 10 of this invention. The electronic system 
P roce88 ° r 1 2 communicating with a memory 14 that stores the programs, documents and other data for 
., Practicing the invention. The processor 12 also communicates with a display 16. a keyboard 18, a mouse 20 and a 
pen 22 via input/output device 24. The keyboard 18. mouse 20. pen 22 and any other interface devices (not shown) 
fjr.T/ 8 ! ^ " L USer J° ?° ntr ° l the ° peration 01 *• electronic system 10. Control of the electronic system 10 is not 
limrted I to the interface devices shown and does not require any particular interlace device. The display 16 may be on 
SJ£Uf^? ne J ,an ?* npul * r or a tablet connected by a tether to a conventional computer or any other type of display 
L™!!L / ^ 8how " " th e system 10 is preferably implemented using a programmed general purpose computer 
^r,!^ T 2 030 a,S ° be fomented using a special purpose computer, a programmed microprocessor 
ZZEEF r ° fy "pessary peripheral integrated circuit elements, an ASIC or other integrated circuit, a 

P?IT£ f ° T ^- .°' rCUit SUC " 88 8 diSCr8,e 8lem8nl circuit ' a P'OQ^able logic device such as a PLD. 

n^h^Te K f °J- J 98nera1, any device °" whicn a Wni,e 6ta,e machine capable of implementing the 
flow charts shown in Figs. 4A-5B can be used to implement the system 10. 
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imp,emen,ed usir * s,a,ic « *~*» ram. 

drive, a hard drive, a flash merS^ or 2 ^1, ^^ *** a Wri,8b,e ^ disk ««* 
dis^ct pontons of a single™™^ * - 1— he either 

wireless finks tone^S^owr^ d "* SB 24 ,0 0,9 PrOCeSsor 12 «* •» «*- « 

•^oranyo^^^ 

Stlon'^ « - -form digto, ink 

42 has been scaled and 32 so ffi?^ JI^SES ?T "? F * The free,on,n di 9 ital ink annoteti °" 
spends with the changeoTZview of fhe *E3hX^nfJ?f T "* anno,a,k5n in me view 
be1ween«hefreeform%^ 

a 2£ Sew to* o^u^ ^ * * 56 - 58 

digital ink annotations 54. 68 arStTJJsSJSS 5 JSSS. b f"y™** "9- 3B shows that the freeform 
document 52 in accordance w°i mJZS^Ji^SS^ ™ ^ retetionsni P s to »• Ejects in the 
has been edited to insert the u« ■.. I^TT: " T. -/r^:. ' ne content of me 52 shown in Fig. 3A 

In response to these edits the text tte^ T^lZl ?£j ?3. J° ^ a PP roxUnatel V ^ "nes ot text at 64. 
freeform diaital ink annr**™'™"™ ™ l™"™ « « now flows across two lines and the "cross-out" 
tionship between -crossJt^o^^ «° COn,inuo to » ^fca. rela- 

ntees that the group of ink strokes m 58 bL ™r^^ under * ,n 9 text The invention also recog- 

margjn note 58 In the margin T^ZSoM^J^rT^ ^ lhe V6rtical P 08 *™ 

line. Additionally, me freefo^ dS^ 

segments of the f reefed d aS 

scaling or otherwise t^^r^^^^J^f'^ ^ m ,he view Th. third step involves moving 
the annotation was created " annotate .n a v,ew and return to the annotated view that existed when 

In the example of Figs. £ iSSSi arXhi „ ' , ° ^ 3re node8 h •» h yP erbo,ic ««"••" 
th^uttrcma.cJiectt^^ 

"arkisancWedtoftecoro^ 
in the view, in the exar^S h FtaT^I^ 

dement in the margZear a„ of L^^f*^^^** ^ * m *™" '° a h «* 

S2S. SS-tSTS^ For example, asshown in Figs. 3A 

is anchored to the span of word^ha^Sl^ 1 C " Cte 68 ««> an arrow 66. The circle 68 

l^ndpore^ 

virtual world The lavoutrt ih« annotation mvolves the freeform digital ink annotation of a door to a room in a 

- arK^s^oUnt^ 

digital ink annotation STe do* as le C^oTth^, ^ ,he l ° 9iCa ' rete,ion8ni P ^tween the freeform 

tatfon rotating with the door ** **" Chan " 8 - ^ 11,6 U8er 8688 tne '™«'<>rm digital ink anno- 

KUrTa^ 

« in a view. A r^ita Zt^T^l ^^T* A " "** ' S 8 ^^"^^1 offset from a position 

.is, of positions" A cnaraXTs iSSS^S^ZTT* * * 3 * 

text are maintained as the text cnanoS A 2 Sf^T Character pos,x,ons between characters in the flow of 
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anchors with offsets. An anchor is either a simple anchor or a composite anchor. An anchored stroke is an anchor plus 
a list of offsets. 

[0033] An embodiment of the method and system of the invention moves the anchor points if the underlying anchored 
objects move. Unanchored segments of the freeform digital ink will stretch to fit between the anchored points. Some- 
5 times adjacent anchor points move apart because the underlying objects move apart: For example, if a text phrase 
reflows from a single line across two lines (as shown in Figs. 3A and 3B). In such cases, the connection between the 
formerly adjacent anchored points can be left out to reduce clutter and any unanchored ink between those points can 
... be removed. Thus, as shown in Fig. 3B. the ref lowed freeform digital ink annotation is split and does not show ink 
. : extending between the two segments, which are on two separate text lines. If anchor objects for a stroke are deleted. 
io then the freeform digital ink annotations can be remapped to the nearest remaining objects in the document. If all 
r anchor objects for the freeform digital ink annotation are deleted then the annotation can be deleted. 
- [0034] The type of the anchors for the freeform digital ink annotation depends upon the shape of the annotation and 
the kind and number of the corresponding objects. One method, detailed below, for computing ink anchors is to select 
the nearest object for each segment in the ink. Alternatively, the type of stroke, as determined from its shape, can be 
used to determine the anchor objects. In another case, a freeform digital ink annotation with multiple anchors may 
cause unwanted distortions when the view changes. To avoid this problem, an embodiment of the present invention 
groups segments of the freeform digital ink annotation together to reduce the number of anchors. 
[0035] As briefly described earlier, an embodiment of the method and system of the present invention uses at least 
three processes. The first process receives freeform digital ink annotations on the view, the second process computes 
a logical mapping between objects in the view and the freelorm digital ink annotations, and the third process renders 
the freeform digital ink annotations on a view that has changed. An optional fourth process selects freeform digital ink 
)/ annotations to navigate to a previous state. 

[0036] The first process includes sensing data points from a pointing device, such as a mouse or a pen, rendering 
the ink strokes by connecting the points on the view, and grouping the ink strokes together to form a freeform digital 
ink annotation. These ink strokes may be grouped, for example, by a simple time threshold. The second process 
receives the freeform digital ink annotations as input and maps between the objects and the freeform digital ink anno- 
tations. The third process updates the view when the document changes. This process uses the mapping to render 
the freeform digital ink annotation on the objects as the view changes. 

[0037] The procedures and the embodiments of the invention described herein assume that the objects do not change 
while the user is inking. However, the method and system of the invention is not limited to applications in which the 
objects being annotated do not change. As an example, in the case of video, the video may be halted while the user 
marks. Alternatively, the video can continue while the user marks and the state of the objects at some point in time 
can be used to interpret the objects and update the mapping between the objects and the freeform digital ink annota- 
tions. 

[0038] The fourth process takes a selection on a view, determines if it corresponds to a freeform digital ink annotation, 
and, if it does, replaces the current view with the view which existed when the freeform digital ink annotation was made! 
This process requires storing an additional mapping from the freeform digital ink annotations to the state of the changing 
view. 

[0039] The present invention recognizes that objects may be built from sub-objects which may be split from each 
*o ^other. One example is a text phrase that is built out of words. If a phrase has a freeform digital ink annotation that 
..circles the phrase and if that phrase reflows across two lines, the invention recognizes this split and may either reflow 
the freeform digital ink annotation by spitting it up into several segments, each of which follows the corresponding 
/: underlying words as they move, or the system may choose to draw lines between matching segments of the circle. 
Alternatively, the freeform digital ink annotation could be replicated at a smaller scale on each individual word. 
[0040] A user can also copy a phrase to another paragraph, which could be considered a form of splitting or as an 
entirely new and unrelated object, without the circle annotation. 

[0041] Conversely, objects can merge into larger entries. For example, if an unannotated word between two anno- 
tated words is deleted then the two annotated words "merge" to form an annotated phrase. The two annotations join 
to form a single annotation of the merged phrase. 
50 [0042] Objects may also disappear entirely. For example, words in a text editor can be deleted. In this case, the 
.-freeform digital ink annotations associated with the object can disappear also, or they can remain at the location be- 
tween the surrounding objects. 

[0043] Objects can also change. For example, a user in a text editor can select an annotated word and replace it 
with a new word. The new word could be considered a a new version of the original object. The annotation on the 
original word may then be associated with the new object. In this case, the annotation may alter its shape to conform 
to the new word, for example, by stretching or shrinking. Alternatively, the annotation could be removed from the display. 
[0044] An embodiment of the method and system of this invention may group ink strokes on text to form a single 
freeform digital ink annotation. By way of non-limiting examples, the annotations can be categorized as circles, under- 
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cross-out annotations are identified bv eloniJt^^ir ^. J bjec,s,ns,dalne ^stroke circle. Underline/ 

These annotations are mappedtt^XS E^Z£2£Z£ T ** « Un * m — ,he ^ 
an .nsertion if the annotations dips into a line of texl t An iroeiZ a nnS^? anno,a,io " ma V *» identified as 

mark appears in the text, for example. brt^S^^^A^?^* * "* Wh ° re ,he insertio " 

margin. The margin note annotations are map^2E?Li* h- 9 ° ,e ' 8 a " annota,io " which is located in a 
of the annotations. Connector annotations a ^en?ifS » o£iS£5? * ^Z*"** ^ hori20ntel <*»«™ 
annotation or by lassos (i.e.. a circle plus a line mSaSZ ZSTTJZZZ" *" ^ aPP ° ar at the end of ,he 
connector portion. Connector annotations are ro^^wSEL ■ C °" nec i or ^notation is split into a circle and a 
occurs at the start of an annotation and tha s^rt ' 'ndependently. The first object is the object that 

oe positions in the text, or - the annotation. These objects £ 

mapped to the circle annotation wh£h. in turn toZJ 9 £*, a^hT ' , 6nd 01 the ,ine <* a lasso fe 

annotation. One end of the line is n^Jr^^lZ t^i^ J 0 ^ 0 " 3 Bn ° ,rom 0,8 taxt to a "^9* 

400 inducing circ.e (or enc.osure) £^^1^ *9- 6 shows a dispfay of a document 

and4l0. insertion annotates 4t 2 and 4l4TriS^ 
™'9*n°te420)andana,rowconnec^ 
in conjunction the des,^^ 

structure of the view underlying the Ueeton^wT^^^Tj"* annotations are "^PP* to the object 
be to hierarchy, object structures s^^^Z ^TZ Wh,le rt » to ^ ^stood that the mapping may 
description Ascribes a mapping of a^gl^^^^^ 

between characters. A texteditor nJZZSl ? t * 9lta ' wk annota,to "« and characters or positions 
have been deieted. the ^^or^Z^rS^T^ 88 the ,e * chan 9 es ^^ersSS 

the text editor stil, maintains a co' e^^S^^^ 

characters. ponaence between the character posrt.ons and some location between remaining 

acters has been deleted: can acauire th« nnehiitn i n ■ a C arac * er Position, a character or a range of char- 
region in the view that corresponds to a XraSeV « Z^^'T^ ,0 3 Character P 08 ^ the 
character position; can acquireTrani ot S^SS^J? ^ P«*«on« that correspond to a 

freetorm digital ink annotaVfon into a £22ZS£E£!Z ^k^T V 'V** 8 ^ "" d 030 a 
character or to a position between characters " ment8 ' such that 6300 stroke corresponds to a single 

Kred°s^ 

SCS^^«r~ or to lo the databa8e> to *** 

pseudocode. ^ embodiment of the invention. The flowchart parallels the following 
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First, classify the group of strokes as circles, underlines/cross-outs, insertions, margin 
notes, or connectors. Notice that this may require splitting some of the strokes, and 
may result in several groups of strokes. Then, 

Sort the new groups of strokes so thai connectors appear after any groups they 
connect to 

For each new group of Strokes 

If the group is a circle or an underline or a cross-out, then 

Split the group of strokes into a list of strokes and characters 
For each stroke and character pair in the list 

Create a new simple anchor corresponding to the character 

Get the region corresponding to the character 

Let anchor position be the center of the region 

Create a new anchored stroke based on the simple anchor 

For each position in the stroke. 

Let offset be position minus anchor position 
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Append offset to the anchored stroke's list of offsets 

End for 

Add anchored stroke to the annotation database 

End for 

Else if the group is an ink insertion into the text, then 

Let character position be the character position where the ink dips into the 

Get the position corresponding to the character position 
Let anchor position be that position 
For each stroke in the group 

Create a new anchored stroke based on the character position 

For each position in the stroke 

Let offset be position minus anchor position 

A PPend offset to the anchored stroke's list of offsets 
end for 

Add anchored stroke to the annotation database 
End for 

Else if the group is a margin annotation, then 

Let character position be the first character position corresponding to the 
vertical extent of the annotation F 8 

l^™ ft ^ P ° sitiom «™*«^ to character position 
Ifthe group is nwsdytote^ P 

Let margin anchor be { character position, flag=left } 
Let anchor position be the left margin position 

Else 

Let margin anchor be { character position, flag=right } 
Let anchor position be the right margin position 

End if 

For each stroke in the group, 

Create a new anchored stroke based on the margin anchor 

For each position in the stroke 

Let offset be position minus anchor position 
Append offset to the anchored stroke's list of offsets 

End for 

Add anchored stroke to the annotation database 
End for 

Else if the group is a connector, then 

For each of starting and ending position on connector 
If position is near an annotation 

Let simple anchor be the anchor for that annotation 
Let anchor position be the current position of the simple 
anchor 

Let offset be position minus anchor position 
Else (position is near an object) 

Let simple anchor be the anchor for that object 

Let anchor position be the current position of the object 

Let offset be position minus anchor position 

End if 

End for 
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Let composite anchor be the anchors and offsets corresponding to the 
starting and ending positions 
s For each stroke in the group 

Create a new anchored stroke based on the composite anchor 
For each position in the stroke 

Let offset be position minus starting position 
io Divide offset by the distance between thestarting and 

ending positions 

Rotate offset around the origin by-angle (ending position 
minus starting position) 

is Append offset to the anchored stroke's list of offsets 

End for 

Add anchored stroke to the annotation database 

End for 

20 End if 

End for 

[0050] The control routine shown in Figs. 4A-4F starts at step S100 and continues to step S102. In step S102, the 
control routine groups the ink strokes into annotations such as circles, underlines/cross-outs, insertions, marginalia or 
connectors. It is to be appreciated that this may require splitting some strokes and may result in several groups of 
strokes. The control routine then continues to step S104 where the annotations are sorted so that connectors appear 
after any group to which they are connected. The control routine then continues to step S106 where the control routine 
initiates a loop that iteratively processes all of these annotations. The control routine determines if the current annotation 
is a circle, an underline or a cross-out. If in step S106, the control routine determines that the current annotation is a 
circle, an underline or a cross-out then the control routine jumps to step S116. If. on the other hand, in step S106 the 
control routine determines that the current annotation does not. include a circle, an underihe or a cross-out then the 
control routine continues to step S108. In step S108. the control routine determines if the current annotation is an ink 
insertion. If, in step S108, the control routine determines that the current annotation is an ink insertion then the control 
routine jumps to step SI 38. If, on the other hand, the control routine determines that the current annotation is not an 
& ink insertion, then the control routine continues to step S110. In step S110. the control routine determines if the if the 
current annotation is a margin annotation. If, in step S110. the control routine determines that the current annotation 
is a margin annotation then the control routine jumps to step S158. If. on the other hand, in step S 110, the control 
routine determines that the current annotation is not a margin annotation then the control routine jumps to step S186. 
[0051] After completing one of the processes of Figs. 4B-4F, which will be explained in detail below, the control 
routine continues to step S112 where the control routine determines if more annotations need to be processed. If in 
step S112, the control routine determines that more annotations need to be processed, then the control routine returns 
to step S106. If, on the other hand, the control routine determines that no more annotations need to be processed, 
. . then the control routine continues to step S11 4 where control is returned to the environment in which the control routine 
of Figs. 4A-4F operate. 

[0052] At step S1 1 6, the control routine splits the annotation into a list of substrokes each with a corresponding single 
character. Thus, the control routine splits the annotation into substroke/character pairs. The control routine then con- 
tinues to step S118, where the control routine initiates a loop to iteratively process each substroke/character pair in 
the current annotation. In step Si 18, the control routine creates a new simple anchor that corresponds to the current 
character. The control routine then continues to step S120 where the control routine acquires the region in the view 
that corresponds to the current character. The control routine then continues to step SI 22 where the control routine 
sets an anchor position to the center of the region in the view. The control routine then continues to step S124 where 
the control routine creates a new anchored stroke that is based upon the simple anchor and continues to step S126. 
In step St 26, the control routine sets an offset to the position from the substroke minus the anchor position and con- 
tinues to step S128. In step S128, the control routine appends the offset to the anchored stroke'6 list of offsets and 
& continues to step S130. In step S130. the control routine determines whether more positions exists in the current 
substroke. If the control routine determines in step S130 that more positions exists in the current substroke, then the 
control routine returns to step SI 26. If. on the other hand, the control routine determines that more positions do not 
exist in the current substroke then the control routine continues to step S132. In step S132, the control routine adds 
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corresponding to the current character pos^ ^^f . W * 9re ,he con,r5>l rou,ine ^'es «he position 

Position is set to the current character So"' "f"" ! ° S,e " 5142 «•"• -EE 

™™ ^iatesalo^^ conimues to step S144 where the control 
stroke based upon the current character positicJ 144 f th « con,ral ™tine creates a new anchored 

8148. the contro. routine sets an offset toTe current *aS S ™tine continues to step S146. In step 

continues to step S148. In step S1 48 the SntoSSiSf ^ F 1 Uom lhe s,roks minus thB an <*or position and 

cc^tinuestostepSISO. InsteTsVw 'the^ 
If .in step 8150. the control rautofdete^ 

returns to step S146. If. on me oft£hS^££S ST^ I"* ^ CUaent ,nen the "EE 
-nthecurrentstroke then ^oan^^^^^B^^ST^ *" "° mow P 08 * 0 " 8 
stroke to the annotation database and the controToSir^ 

determines whether more strokes exist in mTZen^ ^ S154 ' «» routine 

more strokes exist in the current annotation tZZto e ^%™l T" ^ *° COn,ro1 rouUne Ermines that 
contro. routine determines, in step S154. St ^more^s ^SZZT" * &&P 8144 "' ° n lhe *e 
continues to step S112. ° re s,rokes ex,sts ,n »• cw «»nt annotation then the control routine 

the. left and right margin positions that corril^^ 

the control routine determines *^ttE^Z^&£ FT*" » step 8168. In step Sl£ 
routine determines that the annotation is mostly toXT^ h J J« " ° f *" ,eft lf > h s,e P S162. the control 

S1 64. If. on the other hand, the contS raS dSe^rllloK ""'O^ ^ 00n,rel continues to step 

the control routine jumps to step SieT.n sCsiS iTEE * ^ ^ ^ ^ ** le " *S 

r«rt.ne continues to step S166. lnstepS166theccrS 

continues to step S172. °' touXme 88,8 an an *or position to the left margin position and 

tKtrcJrS^^^ 

control routine initiates a loop to pKctm^S^X^T T and con,inues to 8, °P S1 72. In step 8172. the 
a new anchored stroke based u^b^^^L^- anno a,ion - ln 8,8 P *e control rou!ne crea 2 
sets a„ offset to the position frcJTtte 3 m TsZ^° "? 8174 * 8,6P S174 ' *° cortrol 
S176. the control routine appends the offset to ^ f *? continues «° step S176. In step 

S178. the contro. routineTetermines whetne? SSTS3E?S2t T Mu&8 «° S,7a 

routine determines that "K»po.Won8^S^^^? cu ^^ "^P S178. the contS 
he other hand, in step S178. the control roiVc^ZSe^^S ^ '° U,ine re,UmS l ° "«* S174 - »• « 
the control routine continues to steps S180 insten SilH™ ?^ positions exists in the current stroke then 
database and continues to step SlK Instep^e^^^ 

current annotation. If. in step 8*82. "e^21?£2^ t d,,B ^ mwe *«*•■ exists in the 

then the control routine returns to step s^f ™Z oSSS Tl T? ***** h m ° CUTOnt annotation. 

^sasssr cur ™ tk ^ ^ ^sr^.isss assist 

-Cs^e^^ or ending posits h . e current 

near another annotation or an object If in steo Sie^ „ ' * the Stertin9 3,111 endin 9 positions are 

ano^er annotation then the controlle c^Z tl^eXTT^T ** CWrent ^ « "« a ' 

step .8188. that the position b not near anc^^S^^^. , T COntro ' Ermines, in 

818B. the contro. routine eets a simple anchor iSTSSSTi,!! T COn,inUeS f ° 8te P 8192 *ep 

step S 190. the control routine sets an ^!^^^T^- ann, ^ ,0n andco " ti "« 8 * *> «««P S190. In 
step 8196. In step 8192. the control r^Z^^l^^l^T f ,h ° 8hnf>te continues to 

starting or ending position and the control Sne ^Sl^^^f^ ** m ^ whleh is near 1,18 cw ^« 
anchor posHion to the current position of f"* 81 ^ ,he ^ rou,ine 8 * t8 

control .utinesetsan offset - 8 current posUf^ 
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SI 98. the control routine determines whether more starting or ending positions exist in the current annotation. If, in 
step S198, the control routine determines that more starting or ending positions exist in the currenl annotation then 
the control routine returns to step S186. If. on the other hand, the control routine determines, in step S198. that no 
more starting or ending positions exist in the current annotation then the control routine continues to step S200. 
[0057] In step S200, the control routine initiates a loop to process all strokes in the current annotation. In step S200. 
the control routine creates an anchored stroke having a composite anchor based upon the anchors corresponding to 
the starting and ending positions and continues to step S202. In step S202, the control routine sets an offset to the 
current position in the current stroke minus the starting position and continues to step S204. In step S204, the control 
routine divides the offset by the distance between the starting and ending positions and continues to step S206. In 
step S206. the control routine rotates the offset around an origin by an angle equal to the ending position minus the 
starting position and continues to step S208. In step S208, the control routine appends the offset to the anchored 
stroke's list of offsets and continues to step S210. In step S21 0, the control routine determines whether more positions 
exists in the current stroke. If, in step S210, the control routine determines that more positions exists in the current 
stroke then the control routine returns to step S202. If, on the other hand, the control routine determines, in step S210, 
that no more positions exists in the current stroke then the control routine continues to step S212. In step S212 t the 
control routine adds the composite anchor to the annotation database and continues to step S214. In step S214! the 
control routine determines it more strokes exists in the current annotation. If, in step S21 4, the control routine determines 
that more strokes exists in the current annotation then the control routine returns to step S200. If, on the other hand, 
the control routine determines that no more strokes exists in the current annotation, then the control routine continues 
to step S216 where the control routine returns to step S112. 

[0050] Figs. 5A-5B shows a control routine for rendering freeform digital ink annotations on a changed view. The 
flow chart parallels the following pseudocode. 



Using the annotation database, get the list of all anchored strokes 
For each anchored stroke. 

Extract the anchor and the list of oflfeets 
Let stroke be a new stroke 
If anchor is a simple anchor 

Determine if the character or character position underlying the anchor 
has been deleted 
If not, then 

Let anchor position be the position of that anchor 
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For each offset, 

Let new position be offset + anchor position 
Add new position to the stroke 

End for 

End if 



Else 



io ?i tem ? ine * the character or character positions underlying the 1* and 

2°° anchors has been deleted 
If both have been deleted, then 
do nothing 



ss 



Else 



rof&tf 115 P ° Siti0D ^ P ° Siti0n of anchor > P ,us 

Let ending position be the position of the 2 nd anchor, plus the 
2 onset 

20 For each offset, 

Let new offset be offset rotated byangle (ending 

position minus starting position) 

Multiply new offset by the distance between the starting 

25 and ending positions 

Let new position be new offset plus starting position 
Add new position to the stroke 
End for 



End if 

End if 

Render stroke on the view 
End for 



mines whether the «^£2£ J3£?S SSST ? P . 1 ^ the ^ roUti ~ deter - 
anchor then the centre, ^neSn^ 

S308, that the anchor fe not a Jf . J ' other han * the con,rol n*"™ determines, in step 

mines that the character or cSe"S S5^^' ' 1 831 °' ^ C ° n,r °' ,OUline deter - 

reposition to «he stroked ** P S31 * »• °°ntro. routine adds the 

more offsets for the anchor If toaZ Sis , * I™' lh * C ° n,f ° l ,OUtine dete ™ in «* Aether there are 

refine returns to ^3^4 if 

the controi routine jumps to step^io l^SS^S^T? determines J hat ,hero are *» <*•«• then 
tostep S322 In steo sa» th.!.™.~i . ^ . . COn,ro1 rout,ne ,endera «"e stroke on the view and continues 

the control ro^nTr^Tte^ e,roke8 in *• annotatton then 

are no more anchored Ttrokes t ^^SSiTdS^' .l 00 "" 0 ' rOUtine • de,enrnines in 6te P that there 

een aeieiea. if, in step S326. the control routine determines that both of these character or character 
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positions have been deleted then the control routine returns to step S320. If, on the other hand, the control routine 
determines, in step S326, that the character or character position underlining the first and second anchors have not 
been deleted then the control routine continues to step S328. In step S328, the control routine sets the starting position 
to the position of the first anchor plus the first offset and continues to step S330. In step S330, the control routine sets 
the ending position to the position of the second anchor plus the second offset and continues to step S332. In step 
S332, the control routine sets the new offset to the offset from the anchored stroke rotated by an angle equal to ending 
position minus starting position and the control routine continues to step S334. In step S334, the control routine mul- 
tiplies the new offset by the distance between the starting and ending positions and continues to step S336. In step 
S336, the control routine sets the new position to the new offset plus the starting position and continues to step S338. 
In step S338. the control routine adds the new position to the stroke and continues to step S340. In step S340, the 
control routine determines whether more offsets exist. If, in step S340. the control routine determines that more offsets 
exist then the control routine returns to step S332. If, on the other hand, the control routine determines, in step S340, 
that no more offsets exists then the control routine returns to step S320. 

[0061] Freeform digital ink annotation is intended to mean a visible ink stroke of arbitrary shape made with a pen, a 
mouse or other pointing device that persists over time. A pen based input device is intended to include a pen tablet, 
a pen display, a pen display tablet, a pen computer anoVor a touch screen. The systems and methods of this invention 
may infer some meaning from the shapes or the morphological characteristics of ink strokes but do not remove the ink 
stroke once they have been recognized. 

[0062] While the detailed description refers generally to electronic systems, it is intended to be understood that the 
method and system of the present invention may be applied to any electronic book and document reader that displays 
changing views. Electronic book and document readers include any electronic handheld device that can be accessed 
with a pointing device such as a pen and that accepts freeform digital ink as input. Electronic book and document 
readers may also include other additional features without limitation. 

[0063] It is intended to be understood that the term text* is intended to include text, digital ink, audio, video or any 
other content of a document, including the document's structure. An object is intended to be any component of a display 
such as, by way of non-limiting example, a character, a word, a phrase, a sentence, a paragraph, a line, a section, a 
chapter, a portion of an image that is identifiable by a computer vision system in a single image or in a stream of images 
as in a video, etc. 

[0064] It should be understood that a change in the structure of a document is intended to include inserting, deleting, 
copying and moving of objects or the like. It is also intended to be understood that a change in the layout structure of 
a document includes reflowing of objects, resizing of objects, rotation of objects, stretching of objects, distortion of 
objects, morphing of objects or the like. 

[0065] While this invention has been described in conjunction with the specific embodiments outlined above, it is 
evident that many alternatives, modifications and variations are apparent to those skilled in the art. Accordingly, the 
preferred embodiments of the invention as set forth above are intended to be illustrated and not limiting, various chang- 
es may be made without departing from the spirit and scope of the invention. 



Claims 

L A method for maintaining an indication of a logical relationship of at least one freeform digital ink annotation to at 
least one object in a first view as the first view changes, the indication including a layout relationship of the at least 
one freeform digital ink annotation to the at least one object, the method comprising: 

mapping the logical relationship of the at feast one freeform digital ink annotation to the at least one object 
based upon the layout relationship of the at least one freeform digital ink annotation and the at least one object 
in the first view; 
generating a second view; 

rendering the at least one freeform digital ink annotation in response to the generation of the second view, the 
layout relationship of the at least one freeform digital ink annotation and the at least one object in the second 
view based upon the mapped logical relationship of the at least one freeform digital ink annotation to the at 
least one object in the first view. 

2. The method of claim 1 , wherein the second view is generated in response to at least one of a change in the structure 
of a document and a change in the layout of the document. 

3. The method of claims 1 or 2, wherein mapping the logical relationship comprises: 
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4. The method of claim 3, wherein: 

5. The method of any of claims 1 -4, wherein: 
the at least one object comprises at least two objects- and 

- :=dn^ 

o,meat,east<,efree,orrdSi^ 

portion of the freeform dX^^^ each substroke corresponding to the 

of the plurality of substrokes based ^SLl « 1? . ^ ,eCt> and by Creati " 9 an anchor for 
one object in the first view ^ lBy0Ul relat,onsn 'P of each °» the plurality of substrokes to at least 

splitting the at least one fteetorr^ iS iTmXE?.^ 1 ^ ^ 0b|eCtS ^ S8parated ' *• processor 
<-formd ig «a,,kan^^ 

10. The system of any of claims 7-9. wherein: 

the at least one object comprises at least two objects and 

annotation in the firs^eT ^ "* 3 '° Ca,ion ° f at ,east 008 ,ree, °™ digital ink 
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12. The system of any of claims 7-11 , wherein: 

the at least one freeform digital ink annotation includes at least two f reeform digital ink annotations and the at 
least one object includes at least two objects, each object corresponding to one of the at least two f reeform 
digital ink annotations; and 

the processor merges at least two ol the at least two freeform digital ink annotations into a single freeform 
digital ink annotation in response to at least two of the at least two objects being merged. 
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2 Ihe Problems 
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£i this section we present o<nwnbej>Df problems that stand in thg »ny nf m. f ^ m 
Each prob em is described with olew concrete ewmples. end thei/ ovenues ol invesfantSS^ 
Qnot may teod to o solution ore presented in subsections. * i — ^ 



408^ 

2.1 Providing Bjformotion(occess on mobile devices is hard 
Specific exomples of ftis.ffe*tem-j^f" v ~' 410 



^•404 
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Not being able to get the information you need when you're owoy from your desktop 
Woiling for Web poges to download over o slow modem or wireless fink. 

One promise of pcrtoWe computers is nformotion occess anytime, anywhere. People hove 
become refiont on continuous, high-speed, low-cost networks ot their desktop, yet portable 
computers ore rarely connected to such networks. Caching is o technique that mote 
412 portobfe computers less dependent on ubiquitous low-cost networking. 

Vw 7T^r to /^n^414 

wchiraps an ancient computer science techniquefthot stores a copy of frequently 
referenced nformotion near at hand. Although it is a simple concept, o number of subtle 
problems arise in practice. 



vX*fi Caching has adtfilionol indications' for mobile devices. When connecting over wireless 
4^* iVflT^ cache misses not only take o relatively long lime to process, but also may 



^^l^^netwa 



in expensive fees for network occess. Moreover, when operating disconnected from a 
network, cache misses may stop users from continuing with their work. 

LI Renwving documents from local storage 



420 



(Researchers have been investigating new replacement pofctes thot better predict future 
reference! One avenue is to ouqment computation! predictions with dired user 



41 o interactions, on the assumption thot people often know which mlormction is most 
important to keep in bond. Ihe TeJeWeb [Schl96j and Uowgfi [U!jeberg95] mobile web 
browsers allow users to lock documents into the coche. One severe problem with o user- 
be! locking interfoce is thot over time more and more documents become locked 
because users tend to lock more then they purge. In this cose, caching becomes much less 
effective because locked, but no longer relevant, documents con take up o targe port of o 
cache. 
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Jnother>pprooch is to ask (he user which documents to discord instead of which 

cumenls to keep [Goldberg93]. For example, when coche replacement is necessary the 
system might use o 
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